Papers with vision-language task
LANS: A Layout-Aware Neural Solver for Plane Geometry Problem (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing neural solvers take GPS as vision-language task but lack layout awareness . Existing models are criticized for complex rules and poor adaptability . |
| Approach: | They propose a layout-aware neural solver called LANS that integrates two modules to solve GPS. |
| Outcome: | The proposed solver outperforms existing neural and symbolic solvers on two datasets. |
Multi-modal Action Chain Abductive Reasoning (2023.acl-long)
Copied to clipboard
Mengze Li, Tianbao Wang, Jiahe Xu, Kairong Han, Shengyu Zhang, Zhou Zhao, Jiaxu Miao, Wenqiao Zhang, Shiliang Pu, Fei Wu
| Challenge: | Existing models for Abductive Reasoning are limited in their ability to infer the most plausible explanation of incomplete known phenomena. |
| Approach: | They propose a vision-language task that aims to imagine the most plausible event by spatio-temporal grounding in past video and infer the hypothesis of subsequent action chain layer by layer. |
| Outcome: | The proposed model outperforms existing video-language models in terms of effectiveness on the proposed dataset. |
VD-BERT: A Unified Vision and Dialog Transformer with BERT (2020.emnlp-main)
Copied to clipboard
| Challenge: | Prior work focused on attention mechanisms to model complex interactions in visual dialog . a new framework for visual dialog is based on pretrained BERT language models . |
| Approach: | They propose a framework for a vision-dialog Transformer that leverages pretrained BERT language models for Visual Dialog tasks. |
| Outcome: | The proposed framework achieves the top position on the visual dialog leaderboard without pretraining on external vision-language data. |
Modeling Coreference Relations in Visual Dialog (2021.eacl-main)
Copied to clipboard
| Challenge: | Visual dialog is a vision-language task where an agent needs to answer a series of questions grounded in an image based on the understanding of the dialog history and the image. |
| Approach: | They propose two soft constraints that can improve the model’s ability of resolving coreferences in dialog in an unsupervised way based on linguistic knowledge and discourse features of human dialog. |
| Outcome: | The proposed model achieves state-of-the-art performance on the VisDial v1.0 dataset without pretraining on other vision language datasets. |
Task-Aware Resolution Optimization for Visual Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing visual large language models pre-assume a fixed resolution for downstream tasks, leading to sub-optimal performance. |
| Approach: | They propose a formula to determine the optimal resolution for a given vision-language task . they then propose 'parameter-efficient' fine-tuning technique to extend the visual input resolution . |
| Outcome: | The proposed method is based on rigorous experiments on vision-language tasks. |